Comprehensive instruction for test automation on the example of the product company

Tickets Travel Network (Tickets.ua) is a company with a large number of products. In 2016, the company was faced with the task of test automation due to the high cost of regression time — an average of 100 person-days, for which Roman was invited to the position of Middle Automation Engineer. Initially, an automation plan was built. This plan would work for most product companies, as in service companies, automation is implemented custom and based on different metrics of diversified projects.

Let’s briefly look at the Stages Of Test Automation:

1. Evaluation of product and process data in the company.
2. Determining priorities, actions, and deadlines for test automation.
3. Preparing a test department.
4. Implementation of automation by area.
5. Evaluation of implementation results.
6. Plan for future improvement.

1. Evaluation of product and process data in the company

When it comes to a product company, it is essential to evaluate:

• number of services (websites, apps);
• number of corporate solutions;
• number of company products;
• number of developers in the team;
• number of testers;
• number of servers and networks;
• types of inter-service communications;
• availability of manual and automatic tests;
• time for manual and automatic tests.

When I came to Tickets.ua, the picture was as follows:

• up to 15 test engineers;
• up to 80 developers in the team;
• 50 sites and localizations with different product sets;
• a number of enterprise solutions with varying combinations of products;
• several mobile applications;
• monolith with +- 3 APIs for different tasks;
• 10 metasearch engines;
• 10 payment types in different combinations on different sites;
• lack of unit and integration tests;
• absence of test cases;
• no test management system (except for the checklist).

What to do with this data?

2. Determining priorities, actions, and deadlines for test automation

Prioritizing is a crucial step that will show:

• Whether automation of testing is required in your company or on your project;
• What areas of the business need to be automated;
• What levels of automation are required in these areas;
• How to introduce automation (by transferring part of the tasks to developers, by training current QA, by hiring a new team);
• How much time will test automation take;
• What results do we expect to achieve after the work pool is complete.

In order to determine my priorities at company, I started with communication with the team: Team Lead testers and PMs of certain product areas. As a result, I was able to understand which testing processes were the most problematic and which products to automate first.

Priorities were set as follows:

1. automation of airline ticketing for key destinations based on business intelligence;
2. automation of rail transport ticket purchase;
3. automation of all other retail directions;
4. automation of testing in corporate directions;
5. automation in the area of redirect from metasearch.

Not always priorities may coincide with the objectives of the business, but in this case, the most profitable areas were the most important in terms of test automation.

For the first priority on the most profitable sites of the company, I prepared a document describing the areas and elements where data attribution is required and what kind. I put this document in the tasks for FE developers. As a result, it was easier to find the same elements for testing scenarios because of the duplication of use of existing components on most websites. But it did not reduce the number of scripts that much. Next, we needed to do the same for all B2B and B2C sites in 5 different areas and all product localizations. Here it became clear that the process by itself would be too long and not as productive and long-lasting as we would like it to be. So I decided to train all test engineers in automation.

My contract terms allowed for training; within the product area of the company, such automation is necessary, and the management confirmed such a decision. Moreover, it was possible to conduct training during working hours.

3. Preparing a test department

Depending on priorities, company capacity, budget, and other criteria, the testing department is optimized by:

• expanding the existing team with automation engineers;
• transferring testability improvement tasks to the development department;
• increasing the skills of the existing team through internal or external training.

At Tickets.ua, I could clearly see that the existing testing department staff could automate their work.

The training was chosen based on a number of criteria:

• on which stack I can teach automation;
• for which stack there are enough materials on corporate and personal training;
• what is the situation in the market of engineers on the specified stack;
• what tools are flexible, available, and stable in results.

First of all, the direction of Java was chosen, the next level was Selenium — Selenide, then JMeter, Postman, and Rest Assured.

As part of the course program, I gave testers:

• Java basics and OOP;
• Git;
• automation and UI patterns with Selenide;
• API testing basics with Postman and Rest Assured;
• performance testing with JMeter.

The structure of the training

Lectures → homework → practical work. The first course was conducted for a group of 14 people. Then, to improve the assimilation of information and more productive work in the group, the number of participants was reduced to 4-5: equal to the number of front-runners and more meticulous.

The specifics of the training were to devote time outside of working hours and responsibilities to the material and to consolidate the skills in individual work. As part of the homework, the processes could be set up already on the company’s “working” products. The presentation of the material was also combined and varied from class to class, and there were also materials for self-study to improve memorization activity.

4. Implementation of automation by area

To carry out automation you need to go through 3 key steps:

• Prepare the basis for automation.
• Plan and distribute processes.
• Roll out the process.

Base automation on Tickets.ua

To provide automation, it was necessary to prepare a technical core. As such, I chose Java modules: core, core_ui, core_api, avia, railway, insurance. They form a base that is easy to maintain and develop in the future.

All modules were distributed among test engineers for individual work and writing autotests. Each area can be monitored through Merge requests, which allows you to eliminate testers’ errors promptly, give advice and improve the core for re-runs.

Checking clicks from metasearch engines

A significant part of the engineers’ work was spent on checking the conversions for certain routes to the company’s sites from metasearch engines: Aviasales, Momondo, and Skyscanner.

For this direction, we chose to test using the existing API, considering the diversified parameters. This approach was justified by the presence of anti-bot protection, which made it difficult to use only UI testing.

A JSON file with metasearch engines, service criteria, routes, and target sites for redirections was compiled to gather all the parameters together. We manually corrected the data and used it to create validation scripts.

The scenarios included checking:

• redirection to the desired site;
• time required to process the request;
• availability of using payment methods on the sites.

This approach has streamlined the process and reduced the time required to check all options within the scenarios.

Product testing

After the training and completion of the project to test transitions from metasearch engines, the company already had 8 junior engineers who could conduct tests somewhat independently on their products. But there was still an open interaction issue between the products and the engineers.

In order to simplify the process of testing products, in the admin panel were prepared “assistants”:

• to change the system configuration;
• registering users with certain data criteria;
• cancellation of reservations, etc.

In addition, I ran paired testing sessions between engineers: product owners, and “guests.” The session lasted 1-2 hours, with at least 2 engineers. The session was divided into two parts: presenting the initial data on the product to the “guest” and testing the product by the guest.

Thus, it was possible to completely change the engineers’ vision on product testing, eliminate duplication of processes, and get a new picture of the process from the outside.

Implementation of a test management system

The company had no test management system; only a basic checklist in Google Docs was available. Previously the company had already attempted to implement one, but unsuccessfully. At the same time, the team leader believed that such a direction did not work in the testing situation of Tickets.ua.

But I decided to offer options for a solution. Given the lack of budget, I chose free and maximum low-cost systems. They could cover the minimum set of functions, as well as cover the API. But given the rejection of this idea by the team leader and the engineering team, the system of automatic publication of regression states by product for release was never implemented.

5. Evaluation of implementation results

The result, which I evaluated in a year of work, was very figurative. Because at the time, the goal was not metrics but a set of automation implementations.

Among the indicators that could be seen immediately:

1. The kernel on Java modules is flexible, easily optimized, and augmented. Engineers have their own field of work on autotests, which can be constantly improved, saving a lot of time.
2. The automation of the transition check has reduced the time to a figure of 4 days for 8 engineers. After automation, in 1 day, 2-3 engineers can check locations and other details in searches.
3. The guest testing system allowed not only to implement automation for each product but also to eliminate duplicate chains or steps, for example, in-system login testing. Moreover, domain knowledge of related products has improved, allowing you to look at testing from a different perspective or take ready-made blocks from other products.

Other results: Most engineers who received my training now work as automated testers. This means that the skills gained are more than enough to perform automation tasks.

6. Plan for future improvement

Retrospective is important in every project. For a test automation project, especially if it is handed over to the team on its own. At Tickets.ua, I highlighted such opportunities for a future project.

Transferring part of the automation to developers on the Shift Left and Shift Right model.

Thus, developers could independently use the result of an autopilot, taking into account the tech stack on each level: the backend and the frontend.

• Front-end automation should be done with Playwright or Cypress, or Puppeteer. This will simplify the verification of sent requests, capture responses from external services, and verify more UI components.
• In Back-end, focus on the tech stack being used so that the developers could fix it directly. The focus should be on the ability to lock an object, pick it up at a local startup, and run it in Docker.

Improving CI/CD level operation

• Introduce static code analysis: Quality gate for each change entry in GIT, taking into account the speed of static analyzers and the speed of problem alerting.
• Run unit and integration tests.
• Approve code review with multiple levels of approval by more than one engineer.
• Introduce a dynamic run of the environment on Pull/Merge Request with smoke and sanity test suites for faster system investigation and the possibility to fix it yourself.
• Maintain the DEV and SIT environment in stable working conditions.
• Increase the frequency of laying out a new version of the system, albeit with the fewest changes, to reduce the number of backward compatibility errors.

Start test automation from scratch

Recap: recommendations for implementing test automation in a product or service company:

1. Test automation is not always a really valid step. To understand the relevance, it is important to compare one test’s duration (and frequency) to the time of building the automation of such testing.
2. Test automation is not required on products at the PoC or MVP stage. It is more important to release raw product versions more often and get maximum feedback from real users.
3. Test automation should be performed where it is possible to maximize stability and productivity. API automation is more stable and affordable than UI, front-end UI automation is less expensive and more stable than mobile native automation, and pixel-perfect automation are sufficiently expensive and not as objective.
4. Test automation is always required for processes related to databases and their processing. Because autotests avoid errors made by manual testers when entering data, they can optimize a significant portion of the testing department’s work time.
5. The cost of test automation is calculated as the cost of one automated test compared to the cost of conducting such a test in manual mode. That is, it is the amount we spend on automators / the number of such tests in ratio to the amount we spend on manual testers / the number of such tests.
6. The testers, and all technical employees, need to know about the company’s products. This will help to understand where there is an overlap of processes and how processes on different products are not duplicated but optimized and increased.
7. Test engineers should always work on their expertise: in frameworks, techniques, types of cross-service communication, and architecture. This will broaden your understanding and understanding of how projects are developed and give you a better understanding of what should be automated, when it should be automated, and what tools it is better to do it with.